Three-dimensional stereoscopic projection on complex surfaces

ABSTRACT

A method of displaying. The method includes using a system including three-dimensional stereoscopic projection equipment to project an image onto a complex surface of a physical object disposed in an area. The complex surface includes at least one curve or angle. The system also includes a device worn by a user or disposed on a mobile device. The method further includes warping the image based on a geometry of the complex surface, tracking a position and orientation of the user or the mobile device, and further warping the image based on the position and orientation.

BACKGROUND INFORMATION

1. Field:

The present disclosure relates generally to augmented reality, and moreparticularly to a technique for projecting three-dimensional virtualimages onto complex surfaces.

2. Background:

In the manufacturing domain, communicating work instructions tomechanics and technicians on the shop floor can be a challenge. Up toforty percent of a technician's time can be spent on non-value addedtasks such as searching for drawings or other job related information.Therefore, it would be desirable to have a method and apparatus thataddresses this issue by increasing an efficiency of a mechanic andtechnician when building devices or performing maintenance on existingdevices.

SUMMARY

The illustrative embodiments provide for a method of displaying. Themethod includes using a system including three-dimensional stereoscopicprojection equipment to project an image onto a complex surface of aphysical object disposed in an area. The complex surface includes atleast one curve or angle. The system also includes a device worn by auser or disposed on a mobile device. The method further includes warpingthe image based on a geometry of the complex surface, tracking aposition and orientation of the user's point of view or the mobiledevice, and further warping and modifying the image based on theposition and orientation.

The illustrative embodiments also provide for a display apparatus. Theapparatus includes a system including three-dimensional stereoscopicprojection equipment configured to project an image onto a complexsurface of a physical object disposed in an area. The complex surfaceincludes at least one curve or angle. The system also includes a deviceworn by a user or disposed on a mobile device. The apparatus furtherincludes a computer in communication with the system. The computer isconfigured to cause the system to warp the image based on a geometry ofthe complex surface and track, using the device, a position andorientation of the user's gaze or the mobile device. The computer isfurther configured to cause the system to further warp and modify theimage based on the position and orientation.

The illustrative embodiments also provide for a display system. Thedisplay system includes a set of glasses including a right shutter forcovering a right eye of a user and a left shutter for covering a lefteye of the user. The display system further includes a plurality ofcameras oriented to observe the user as the user moves within an areaand to sense a position and orientation of the user in the area. Thedisplay system further includes a projector configured to project, ontoa complex surface, a right image for the right eye and a left image forthe left eye, the complex surface comprising at least one curve orangle. The display system further includes a computer system incommunication with the set of glasses, the plurality of cameras, and theprojector. The computer system is configured to cause the projector toproject the left image when the left shutter is open and to project theright image when the right shutter is open, cause the set of glasses toclose the right shutter and open the left shutter when the left image isprojected, and to close the left shutter and open the right shutter whenthe right image is projected. The computer system is further configuredto warp the left image and the right image to fit the complex surface.Warping and modification is further based on the position of the user inthe area. The left image and the right image are configured such thatthe user will perceive a three-dimensional image from the user's pointof view, projected onto the complex surface.

BRIEF DESCRIPTION OF THE DRAWINGS

The novel features believed characteristic of the illustrativeembodiments are set forth in the appended claims. The illustrativeembodiments, however, as well as a preferred mode of use, furtherobjectives and features thereof, will best be understood by reference tothe following detailed description of an illustrative embodiment of thepresent disclosure when read in conjunction with the accompanyingdrawings, wherein:

FIG. 1 is flowchart of a method for displaying an image, in accordancewith an illustrative embodiment;

FIG. 2 is a block diagram of an apparatus for displaying an image, inaccordance with an illustrative embodiment;

FIG. 3 is a block diagram of a display system, in accordance with anillustrative embodiment;

FIG. 4 is an example of a display system, in accordance with anillustrative embodiment;

FIG. 5 is an example of an image displayed on an instrument panel, inaccordance with an illustrative embodiment;

FIG. 6 is an example of an image displayed on a computer system, inaccordance with an illustrative embodiment;

FIG. 7 is an example of an image displayed on a circuit board, inaccordance with an illustrative embodiment; and

FIG. 8 is an illustration of a data processing system, in accordancewith an illustrative embodiment.

DETAILED DESCRIPTION

The illustrative embodiments recognize and take into account the issuesdescribed above with respect to increasing the efficiency of a mechanicor technician when performing maintenance or building devices. Thus, theillustrative embodiments provide for a stereoscopic projection system.In one example, the illustrative embodiments may use active shutterglasses combined with a projector, along with a computer, to project animage onto one or more complex surfaces of a tangible object during themanufacturing process, the image appearing three-dimensional to a user.

By understanding the pose of the projector in relation to the surfaceand the geometry of the surface itself, the illustrative embodiments mayapply a distortion mask to the projected images, warping them such whenthe images are mapped onto the surface, the integrity of the intendedpicture is maintained. In other words, when the warping process isapplied to the projection, the user may perceive a three-dimensionalimage projected onto the complex surface. The projection will appearcorrect to a user regardless of the user's position in a room, theuser's head position, or possibly other parameters that could affect howa similar, real object would appear to the user in the same context. Theprojection could include new parts to be installed, maintenance to beperformed, tools to be used, a moving image of a series of actions to betaken, written instructions, and many other useful images.

Thus, the illustrative embodiments provide for a method and a system touse stereoscopic projection on complex surfaces to provide the illusionof a three-dimensional image. In a specific example, a mechanic ortechnician may use this projection to view, for example, assemblyinstructions in three dimensions that are spatially registered to thepart and directly projected onto the part. These assembly instructionsmay include a three-dimensional visualization of work instructions.

The illustrative embodiments may be implemented using a number ofdifferent techniques for projecting an image that at least appears to bethree-dimensional. For example, the illustrative embodiments may use aprojector, computer that controls the projector, and a pair ofstereoscopic shutter glasses. When one eye of the glasses is shut, theprojector projects a first image intended for the opposite eye. When theother eye of the glasses is shut, the projector projects a second imageintended for the second eye. The images and shutters alternate morerapidly than the eye can readily perceive; thus, the result when thecomputer properly controls the projector is an image that appearsthree-dimensional to a user.

However, the illustrative embodiments are not limited to stereoscopicshutter glasses. For example, the illustrative embodiments may use colorglasses, polarized glasses, or other techniques to project a twodimensional image that appears three-dimensional.

The illustrative embodiments may improve manufacturing efficiency andmay reduce manufacturing costs. For example, by providing an immersiveexperience and viewing work instructions, models of parts and otherrelevant information, efficiency can be increased relative to a mechanicor technician spending time finding and then reviewing two dimensionalplans that are not registered to the actual device or part being worked.

Attention is now turned to the Figures. FIG. 1 is flowchart of a methodfor displaying an image, in accordance with an illustrative embodiment.Process 100 shown in FIG. 1 may be implemented using a system includingstereoscopic projection equipment configured to project an image onto acomplex surface of a physical object disposed in an area. As usedherein, the term “complex surface” is defined as a real surface of areal object, the surface having at least one curve or angle. A complexsurface may have many objects with many turns, angles, corners, curves,and possibly disjointed features, and may be highly complex (such as acircuit board, for example). The system used to implement process 100may also include a device worn by a user or disposed on a mobile device,such as shutter glasses, color glasses, polarized glasses, or other aidfor viewing the appearance of a three-dimensional projection. The systemmay also include a computer configured to control the projector and/orstereoscopic shutter glasses. An example of such a computer is shown inFIG. 8. The system may also include cameras configured to measure aposition and orientation of a user or a mobile device wearing theglasses.

While the following flow is described as a “process” performing one ormore actions, this “process” is defined as a tangible device taking oneor more steps. For example, a computer may control the projector and/orstereoscopic glasses, a camera may receive and process an image, and aprojector may project photons necessary or desirable to project adesired image. Returning to FIG. 1, the process may warp the image basedon a geometry of the complex surface (operation 102). The process thenmay track a position and orientation of the user or the mobile device(operation 104). In one example, the process may track a position andorientation of the user's point of view or, in the case of a mobiledevice, may track the mobile device.

The process may then further warp and modify the image based on theposition and orientation (operation 106). The process may terminatethereafter.

Process 100 of FIG. 1 may be further varied. For example, a totalwarping of the image may cause the image to appear to the user or themobile device as a three-dimensional representation of a virtual objectas seen from a perspective of the user or the mobile device at theposition. In another illustrative embodiment, the three-dimensionalimage further comprises a moving image, such as a video or moviedemonstrating a particular set of operations and parts used with regardto the physical object.

In the case where the physical object is a part, such as a part of anaircraft, the image further may include written instructions for takingan action with the virtual object with respect to the physical object.In the case where the three-dimensional image is a tool, the movingimage may include using the tool to perform an action with respect tothe part. Other variations are possible. For example, the virtual objectmay be a virtual representation of a second physical object to be addedto the physical object.

FIG. 2 is a block diagram of an apparatus for displaying an image, inaccordance with an illustrative embodiment. Apparatus 200 may be used toimplement process 100 of FIG. 1.

Apparatus 200 may include system 202. System 202 may includethree-dimensional stereoscopic projection equipment 204. Stereoscopicprojection equipment 204 may be configured to project image 206 ontocomplex surface 208 of physical object 210 disposed in area 212. Complexsurface 208 includes at least one curve or angle. System 202 alsoincludes device 214 worn by user 216 or disposed on mobile device 218.The term “complex surface” is defined formally above.

Apparatus 200 also includes computer 220. Computer 220 is incommunication with apparatus 200. Computer 220 may be configured tocause apparatus 200 to warp image 206 based on a geometry of complexsurface 208. Computer 220 may further be configured to track, usingdevice 214, a position and orientation of user 216 or mobile device 218.The computer may be further configured to cause apparatus 200 to furtherwarp image 206 based on the position and orientation.

Apparatus 200 may be further varied. For example, system 202 may befurther configured such that a total warping of image 206 causes image206 to appear to user 216 or mobile device 218 as a three-dimensionalrepresentation of a virtual object as seen from a perspective of user216 or mobile device 218 at the position. In another example, complexsurface 208 may be part of a first part. In this case, computer 220 maybe further configured to cause system 202 to project one or more imagesselected from the group consisting of: first images of assembling asecond part on the first part, second images of disassembly of the firstpart, third images of reworking of the first part, fourth images ofusing a tool on the first part, fifth images comprising writteninstructions for taking an action with respect to the first part, andsixth images for inspecting the first part, and seventh images for ademonstration of use of the first part.

FIG. 3 is a block diagram of a display system, in accordance with anillustrative embodiment. Display system 300 may be, for example, part ofsystem 202 and/or apparatus 200 of FIG. 2. Display system 300 may beused to implement process 100 shown in FIG. 1.

Display system 300 may include set of glasses 302. Set of glasses 302may include right shutter 304 for covering a right eye of a user andleft shutter 306 for covering a left eye of the user.

Display system 300 may also include plurality of cameras 308 oriented toobserve the user as the user moves within area 310 and to sense aposition and orientation of the user in area 310. Display system 300 mayalso include projector 312 configured to project, onto complex surface314 of object 316, right image 320 for the right eye and left image 318for the left eye. The complex surface includes at least one curve orangle. The term “complex surface” is defined above.

Display system 300 may also include computer system 322 in communicationwith set of glasses 302, plurality of cameras 308, and projector 312.Computer system 322 may be, for example, data processing system 800 ofFIG. 8.

Computer system 322 may be configured to cause projector 312 to projectleft image 318 when left shutter 306 is open and to project right image320 when right shutter 304 is open. Computer system 322 may be furtherconfigured to cause set of glasses 302 to close right shutter 304 andopen left shutter 306 when left image 318 is projected, and to closeleft shutter 306 and open right shutter 304 when right image 320 isprojected.

Computer system 322 may be further configured to warp left image 318 andright image 320 to fit complex surface 314. Warping may be further basedon the position of the user in area 310. Left image 318 and right image320 may be configured such that user will perceive three-dimensionalimage 324 projected onto complex surface 314.

The illustrative embodiments described above may be further varied orexpanded. For example, the position of the user may be athree-dimensional position of the user in the area. The complex surfacemay comprises a physical part, such as a part to be assembled,disassembled, maintenance, or otherwise worked-on.

In an illustrative embodiment, left image 318 and right image 320 may bea virtual modification of the physical part. In this case, the virtualmodification of the physical part may be a virtual representation of asecond physical part to be added to the physical part. Additionally,left image 318 and right image 320 further may include virtuallydisplayed written instructions for attaching the second physical part tothe physical part.

In still another illustrative embodiment, left image 318 and right image320 may also include a virtual tool used for attaching the secondphysical part. Additionally, left image 318 and right image 320 furthermay include virtually displayed written instructions for attaching thesecond physical part to the physical part. Left image 318 and rightimage 320 further may include a virtual tool used for attaching thesecond physical part. In this case, computer system 322 may be furtherconfigured to cause a moving image of using the virtual tool to installthe second physical part.

In an illustrative embodiment, the virtual modification may be a movingimage of replacing a virtual component of the physical part.Alternatively, the virtual modification may be a moving image ofremoving a portion of the physical part. Further, the virtualmodification, may be a moving image of using a virtual tool on thephysical part.

Additional variations are also possible. Most generally, any image, setof images, moving images, or sequence of images may be projected onto apart to appear as either two dimensional or three-dimensional imagesthat assist a mechanic or technician to perform some kind of activitywith respect to an object having a complex surface. In this manner, themechanic or technician might not require as much time finding andinterpreting instructions for a procedure to be taken with respect tothe object.

FIG. 4 is an example of a display system, in accordance with anillustrative embodiment. Display system 400 may be an example of displaysystem 300 of FIG. 3 or apparatus 200 of FIG. 2. Display system 400 maybe used to implement process 100 of FIG. 1.

Display system 400 includes at least one camera, but preferably two ormore cameras 402 placed around selected locations in area 404. Area 404may be a room, manufacturing area, or any space defined by a boundarydefined by cameras 402.

Display system 400 also includes set of glasses 406, computer system408, and projector 410. Each of cameras 402, set of glasses 406, andprojector 410 are in communication with computer system 408. Computersystem 408 may or may not be in or near area 404. Computer system 408may be one or more computers, possibly operating remotely.

Cameras 402 may be motion capture cameras. Cameras 402 may sensepositions of markers 412 on set of glasses 406. Thus, computer system408 may take as input changing positions of set of markers 412 relativeto coordinate frame 414 defined for area 404. Computer system 408 thusmay determine a position and an orientation of set of glasses 406.However, cameras 402 may also be used to sense a position andorientation of a human user, a part of a human user (such as the user'shead), or a machine such as a robot or other mobile device. Thus, forexample, a user may observe through additional cameras on the robot orother mobile device, and covered by set of glasses 406, so that user mayperform operations remotely.

In any case, computer system 408 controls projector 410 to project image418 on complex object 420. Image 418 may be an actual two-dimensionalimage. However, when viewed through set of glasses 406 the user willperceive on complex object 420 that image 418 is a three-dimensionalimage. Set of glasses 406 may be stereoscopic glasses, colored glasses,polarized glasses, or some other device for allowing a user to believehe or she is perceiving image 418 as three-dimensional, when in factimage 418 is projected onto complex object 420 as a two-dimensionalimage.

Image 418 may be warped in order to account for movements of the userand also a complex surface of complex object 420. If image 418 were notwarped, then as the user moved about area 404 image 418 would becomedistorted and appear incorrect or, in extreme cases, possiblyincomprehensible. Likewise, because complex object 420 is not a simpletwo-dimensional object (such as a movie screen), image 418 again wouldappear distorted or incorrect, or possibly incomprehensible, if image418 is not warped. Computer system 408 may warp image 418 to accommodateboth distortion effects. Thus, computer system 408 may control projector410 to project a warped image that takes into one or both of movement ofthe user and a complex geometry of complex object 420.

Thus, no matter where a user moves in area 404, and regardless of thegeometry of the surface of complex object 420, the user will perceiveimage 418 as being correct. As used herein the term “correct” means theuser perceives a projected image that appears as a corresponding realobject would appear on complex object 420, taking into account aposition and perspective of the user. Thus, for example, as a user movesabout area 404, the user will perceive a tool (such as a power drill ina non-limiting example) in the same changing orientations as the userwould see had the tool been a real tool in the same position as image418 with respect to complex object 420.

While display system 400 may be used to assist a mechanic or technicianto assemble, disassemble, re-work, maintain, or otherwise use or modifya complex object, including the display of written instructions whilethe work is taking place, the illustrative embodiments are not limitedto this use. For example, the illustrative embodiments may be used togive a guided interactive tour, or may be used for entertainment.Projected images may be in color or may be black and white.

FIG. 5, FIG. 6, and FIG. 7 are all examples of the illustrativeembodiments in use. Thus, each of these Figures show what a user seeswhen a device, such as apparatus 200 of FIG. 2, display system 300 ofFIG. 3, or display system 400 of FIG. 5 projects an image onto a realobject in a manner that a user perceives a three-dimensional image. Eachimage shown in FIG. 5 through FIG. 7 may be produced using proceduressimilar to those described above with respect to FIG. 1 through FIG. 4.Although the images shown in FIG. 5 through FIG. 7 are specific examplesof the illustrative embodiments in use, the claims and other uses of theillustrative embodiments are not necessarily limited to these examples.

FIG. 5 is an example of an image displayed on an instrument panel, inaccordance with an illustrative embodiment. Image 500 appears to be athree-dimensional image of a drill drilling an image of hole into a realobject, which in this case is instrument panel 502. While at firstappearing to be a simple projection, such a projection is quite useful.For example, the exact position of the hole can be projected ontoinstrument panel 502 so that the user need not measure exactly where thehole is to be drilled. A user may then use a real drill to drill a holeat the position indicated. The specific drill to be used, and thespecific drill bit to be used may also be shown or inferred. Writteninstructions, such as but not limited to instructions to control therate of drilling, may also be projected.

As a user moves his or her head to gain different perspectives of image500, the projection will change so that the user perceives image 500 inthe same orientation as if a real drill had been present. Thus, forexample, a user may perceive a side of the drill and a position of achuck on the drill not perceivable except from the side, or anyinformation desired to be projected.

FIG. 6 is an example of an image displayed on a computer system, inaccordance with an illustrative embodiment. In this case, image 600 isan image of a part to be installed into a real computer or an electronicdevice, designated as device 602. In this case, the correct orientationof the part is shown as part of image 600. Additionally, images of thewiring used to install the part are also shown, including images ofwhich wire is to be attached to which corresponding input or output indevice 602. In this manner, a mechanic or technician can easily see howand where the part is to be installed, and how and where the new part isto be wired with respect to device 602. Again, written instructions 604may be projected onto device 602 detailing how the part is to beinstalled. Note that the projection of the written instructions mustalso be properly warped so as to take into account the user's headposition and the complex geometry of device 602. If not properly warped,the written instructions may be unintelligible as the user moves his orher head, or if projected onto a highly detailed surface such as thatshown on device 602.

FIG. 7 is an example of an image displayed on a circuit board, inaccordance with an illustrative embodiment. Image 700 is projected ontothe highly contoured surface of real circuit board 702. Image 700 showsadditional parts and wiring to be added to circuit board 702. From thisimage a technician can quickly see what parts are needed and where andhow these parts should be added. The technician can also quickly see howparts are to be wired. Different wires and different parts can appear indifferent colors, so as to aid the technician in understanding how toproceed. Written instructions may also be projected in order to guideand instruct the technician. Additionally, a moving picture may be shownthat indicates the parts being installed in a desired order or accordingto a desired procedure.

The illustrative embodiments described above may be still furthervaried. Any image helpful for assembling, disassembling, re-working,maintenance, or otherwise manipulating an object may be projecteddirectly onto the object. Written instructions or movies may beprojected. Additional material may also be projected that is notnecessarily relevant to the operation, if desired. If an audio system isprovided, the computer controlling the projection apparatus may furthercause audio instructions or other appropriate audio feedback for thetechnician or mechanic. For example, an audio voice may instruct thetechnician to take a particular action and then assure the technicianthat success is indicated by hearing an audible “click”, or other noise,after taking the particular action. Thus, the illustrative embodimentsmay be varied from those described above, and the specific examplesprovided herein do not necessarily limit the claimed inventions.

FIG. 8 is an illustration of a data processing system, in accordancewith an illustrative embodiment. Data processing system 800 in FIG. 8 isan example of a data processing system that may be used in conjunctionwith the techniques described above. In this illustrative example, dataprocessing system 800 includes communications fabric 802, which providescommunications between processor unit 804, memory 806, persistentstorage 808, communications unit 810, input/output (I/O) unit 812, anddisplay 814.

Processor unit 804 serves to execute instructions for software that maybe loaded into memory 806. Processor unit 804 may be a number ofprocessors, a multi-processor core, or some other type of processor,depending on the particular implementation. A number, as used hereinwith reference to an item, means one or more items. Further, processorunit 804 may be implemented using a number of heterogeneous processorsystems in which a main processor is present with secondary processorson a single chip. As another illustrative example, processor unit 804may be a symmetric multi-processor system containing multiple processorsof the same type.

Memory 806 and persistent storage 808 are examples of storage devices816. A storage device is any piece of hardware that is capable ofstoring information, such as, for example, without limitation, data,program code in functional form, and/or other suitable informationeither on a temporary basis and/or a permanent basis. Storage devices816 may also be referred to as computer readable storage devices inthese examples. Memory 806, in these examples, may be, for example, arandom access memory or any other suitable volatile or non-volatilestorage device. Persistent storage 808 may take various forms, dependingon the particular implementation.

For example, persistent storage 808 may contain one or more componentsor devices. For example, persistent storage 808 may be a hard drive, aflash memory, a rewritable optical disk, a rewritable magnetic tape, orsome combination of the above. The media used by persistent storage 808also may be removable. For example, a removable hard drive may be usedfor persistent storage 808.

Communications unit 810, in these examples, provides for communicationswith other data processing systems or devices. In these examples,communications unit 810 is a network interface card. Communications unit810 may provide communications through the use of either or bothphysical and wireless communications links.

Input/output (I/O) unit 812 allows for input and output of data withother devices that may be connected to data processing system 800. Forexample, input/output (I/O) unit 812 may provide a connection for userinput through a keyboard, a mouse, and/or some other suitable inputdevice. Further, input/output (I/O) unit 812 may send output to aprinter. Display 814 provides a mechanism to display information to auser.

Instructions for the operating system, applications, and/or programs maybe located in storage devices 816, which are in communication withprocessor unit 804 through communications fabric 802. In theseillustrative examples, the instructions are in a functional form onpersistent storage 808. These instructions may be loaded into memory 806for execution by processor unit 804. The processes of the differentembodiments may be performed by processor unit 804 using computerimplemented instructions, which may be located in a memory, such asmemory 806.

These instructions are referred to as program code, computer usableprogram code, or computer readable program code that may be read andexecuted by a processor in processor unit 804. The program code in thedifferent embodiments may be embodied on different physical or computerreadable storage media, such as memory 806 or persistent storage 808.

Program code 818 is located in a functional form on computer readablemedia 820 that is selectively removable and may be loaded onto ortransferred to data processing system 800 for execution by processorunit 804. Computer readable media 820 may be referred-to as anon-transitory computer readable storage medium. Program code 818 andcomputer readable media 820 form computer program product 822 in theseexamples. In one example, computer readable media 820 may be computerreadable storage media 824 or computer readable signal media 826.Computer readable storage media 824 may include, for example, an opticalor magnetic disk that is inserted or placed into a drive or other devicethat is part of persistent storage 808 for transfer onto a storagedevice, such as a hard drive, that is part of persistent storage 808.Computer readable storage media 824 also may take the form of apersistent storage, such as a hard drive, a thumb drive, or a flashmemory, that is connected to data processing system 800. In someinstances, computer readable storage media 824 may not be removable fromdata processing system 800.

Alternatively, program code 818 may be transferred to data processingsystem 800 using computer readable signal media 826. Computer readablesignal media 826 may be, for example, a propagated data signalcontaining program code 818. For example, computer readable signal media826 may be an electromagnetic signal, an optical signal, and/or anyother suitable type of signal. These signals may be transmitted overcommunications links, such as wireless communications links, opticalfiber cable, coaxial cable, a wire, and/or any other suitable type ofcommunications link. In other words, the communications link and/or theconnection may be physical or wireless in the illustrative examples.

In some illustrative embodiments, program code 818 may be downloadedover a network to persistent storage 808 from another device or dataprocessing system through computer readable signal media 826 for usewithin data processing system 800. For instance, program code stored ina computer readable storage medium in a server data processing systemmay be downloaded over a network from the server to data processingsystem 800. The data processing system providing program code 818 may bea server computer, a client computer, or some other device capable ofstoring and transmitting program code 818.

The different components illustrated for data processing system 800 arenot meant to provide architectural limitations to the manner in whichdifferent embodiments may be implemented. The different illustrativeembodiments may be implemented in a data processing system includingcomponents in addition to or in place of those illustrated for dataprocessing system 800. Other components shown in FIG. 8 can be variedfrom the illustrative examples shown. The different embodiments may beimplemented using any hardware device or system capable of runningprogram code. As one example, the data processing system may includeorganic components integrated with inorganic components and/or may becomprised entirely of organic components excluding a human being. Forexample, a storage device may be comprised of an organic semiconductor.

In another illustrative example, processor unit 804 may take the form ofa hardware unit that has circuits that are manufactured or configuredfor a particular use. This type of hardware may perform operationswithout needing program code to be loaded into a memory from a storagedevice to be configured to perform the operations.

For example, when processor unit 804 takes the form of a hardware unit,processor unit 804 may be a circuit system, an application specificintegrated circuit (ASIC), a programmable logic device, or some othersuitable type of hardware configured to perform a number of operations.With a programmable logic device, the device is configured to performthe number of operations. The device may be reconfigured at a later timeor may be permanently configured to perform the number of operations.Examples of programmable logic devices include, for example, aprogrammable logic array, programmable array logic, a field programmablelogic array, a field programmable gate array, and other suitablehardware devices. With this type of implementation, program code 818 maybe omitted because the processes for the different embodiments areimplemented in a hardware unit.

In still another illustrative example, processor unit 804 may beimplemented using a combination of processors found in computers andhardware units. Processor unit 804 may have a number of hardware unitsand a number of processors that are configured to run program code 818.With this depicted example, some of the processes may be implemented inthe number of hardware units, while other processes may be implementedin the number of processors.

As another example, a storage device in data processing system 800 isany hardware apparatus that may store data. Memory 806, persistentstorage 808, and computer readable media 820 are examples of storagedevices in a tangible form.

In another example, a bus system may be used to implement communicationsfabric 802 and may be comprised of one or more buses, such as a systembus or an input/output bus. Of course, the bus system may be implementedusing any suitable type of architecture that provides for a transfer ofdata between different components or devices attached to the bus system.Additionally, a communications unit may include one or more devices usedto transmit and receive data, such as a modem or a network adapter.Further, a memory may be, for example, memory 806, or a cache, such asfound in an interface and memory controller hub that may be present incommunications fabric 802.

The description of the different illustrative embodiments has beenpresented for purposes of illustration and description, and is notintended to be exhaustive or limited to the embodiments in the formdisclosed. Many modifications and variations will be apparent to thoseof ordinary skill in the art. Further, different illustrativeembodiments may provide different features as compared to otherillustrative embodiments. The embodiment or embodiments selected arechosen and described in order to best explain the principles of theembodiments, the practical application, and to enable others of ordinaryskill in the art to understand the disclosure for various embodimentswith various modifications as are suited to the particular usecontemplated.

What is claimed is:
 1. A method comprising: using a system comprising three-dimensional stereoscopic projection equipment to project an image onto a complex surface of a physical object disposed in an area, the complex surface comprising at least one curve or angle, the system also comprising a device worn by a user or disposed on a mobile device; warping the image based on a geometry of the complex surface; tracking a position and orientation of the user's point of view or the mobile device; and further warping and modifying the image based on the position and orientation.
 2. The method of claim 1, wherein a total warping of the image causes the image to appear to the user or the mobile device as a three-dimensional representation of a virtual object as seen from a perspective of the user or the mobile device at the position.
 3. The method of claim 2, wherein the three-dimensional representation further comprises a moving image.
 4. The method of claim 2, wherein the physical object comprises a part, and wherein the image further comprises written instructions for taking an action with the virtual object with respect to the physical object.
 5. The method of claim 3, wherein the three-dimensional representation comprises a tool, and wherein the moving image comprises using the tool to perform an action with respect to the part.
 6. The method of claim 2, wherein the virtual object comprises a virtual representation of a second physical object to be added to the physical object.
 7. An apparatus comprising: a system comprising three-dimensional stereoscopic projection equipment configured to project an image onto a complex surface of a physical object disposed in an area, the complex surface comprising at least one curve or angle, the system also comprising a device worn by a user or disposed on a mobile device; a computer in communication with the system, the computer configured to: cause the system to warp the image based on a geometry of the complex surface; track, using the device, a position and orientation of the user's point of view or the mobile device; and cause the system to further warp and modify the image based on the position and orientation.
 8. The apparatus of claim 7, wherein the system is further configured such that a total warping of the image causes the image to appear to the user or the mobile device as a three-dimensional representation of a virtual object as seen from a perspective of the user or the mobile device at the position.
 9. The apparatus of claim 7, wherein computer is further configured to cause the system to project one or more images selected from the group consisting of: first images of assembling a second part on the complex surface, second images of disassembly of the complex surface, third images of reworking of the complex surface, fourth images of using a tool on the complex surface, fifth images comprising written instructions for taking an action with respect to the complex surface, and sixth images for inspecting the complex surface, and seventh images for a demonstration of use of the complex surface.
 10. A display system comprising: a set of glasses comprising a right shutter for covering a right eye of a user and a left shutter for covering a left eye of the user; a plurality of cameras oriented to observe the user as the user moves within an area and to sense a position and orientation of the user in the area; a projector configured to project, onto a complex surface, a right image for the right eye and a left image for the left eye, the complex surface comprising at least one curve or angle; a computer system in communication with the set of glasses, the plurality of cameras, and the projector, the computer system configured to: cause the projector to project the left image when the left shutter is open and to project the right image when the right shutter is open; cause the set of glasses to close the right shutter and open the left shutter when the left image is projected, and to close the left shutter and open the right shutter when the right image is projected; and warp the left image and the right image to fit the complex surface, wherein warping is further based on the position of the user in the area, and wherein the left image and the right image are configured such that user will perceive a three-dimensional image projected onto the complex surface.
 11. The display system of claim 10, wherein the position of the user is a three-dimensional position of the user in the area.
 12. The display system of claim 10, wherein the complex surface comprises a physical part.
 13. The display system of claim 12, wherein the left image and the right image comprise a virtual modification of the physical part.
 14. The display system of claim 13, wherein the virtual modification of the physical part comprises a virtual representation of a second physical part to be added to the physical part.
 15. The display system of claim 14, wherein left image and the right image further comprise virtually displayed written instructions for attaching the second physical part to the physical part.
 16. The display system of claim 15, wherein the left image and the right image further comprise a virtual tool used for attaching the second physical part.
 17. The display system of claim 15, wherein: left image and the right image further comprise virtually displayed written instructions for attaching the second physical part to the physical part; the left image and the right image further comprise a virtual tool used for attaching the second physical part; and the computer system is further configured to cause a moving image of using the virtual tool to install the second physical part.
 18. The display system of claim 13, wherein the virtual modification comprises a moving image of replacing a virtual component of the physical part.
 19. The display system of claim 13, wherein the virtual modification comprises a moving image of removing a portion of the physical part.
 20. The display system of claim 13, wherein the virtual modification comprises a moving image of using a virtual tool on the physical part. 